Repository navigation
feat(cache): add opt-in scoped semantic generation cache - #56
Conversation
Behind SEMANTIC_CACHE_ENABLED and an explicit route or key opt-in in SEMANTIC_CACHE_POLICY_JSON, managed nonstream chat requests without tools can reuse a complete answer whose last user message has cosine similarity of at least 0.98. Every other part of the request, the model, sampling settings and policy revisions must match exactly, and a hit is refused when numbers, dates, negations or quoted text differ. Embeddings use an explicitly configured, priced model capped at 1,024 tokens and $0.001 per lookup. Entries are scoped to the principal, kept for 300 seconds, 256 per principal and 1 MiB each, in D1 (migration 0026) and the existing media bucket. Hits report semantic-hit provenance with no provider call; zero-content retention bypasses the cache.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | ddffeba | Oct 09 2026, 02:35 AM |
Summary
Adds an opt-in scoped semantic generation cache. With
SEMANTIC_CACHE_ENABLEDoff andSEMANTIC_CACHE_POLICY_JSON={}(the defaults), and for any route or key that isn't opted in, the exact cache and every request are unchanged.provider:modelembedding model that the gateway already has configured and priced.allow_tools: falseandallow_streams: falseare supported. Setting either to true is invalid and turns the feature off.today,latest,weatherandstockbypass the cache.X-MultiLLM-Cache: semantic-hit,X-MultiLLM-Cache-Backend: semantic-d1-r2,X-MultiLLM-Usage-Basis: cache-served,X-MultiLLM-Provider-Calls: 0andAge.hitmarker, so accounting, reservations and idempotency settle exactly as for an exact-cache hit.no-store,no-cache,max-ageandX-MultiLLM-Cache: refreshare honored.semantic_generation_cache(migration0026): scoped digests, a bounded vector, expiry, an integrity hash and a body pointer.multillm_mediabucket undersemantic-cache/.cache_response_allowedclassification are stored.semantic_cache_schema_missingbefore any embedding or generation.Details are in
docs/semantic-generation-cache.md.Shared registrations
tests/test_semantic_generation_cache_worker.mjsintest:worker.worker/semantic-cache-d1.mjsandworker/semantic-generation-cache.mjsin the edge module inventory.SEMANTIC_CACHE_ENABLEDandSEMANTIC_CACHE_POLICY_JSONin the Container env keys.semantic_generation_cacheinREQUIRED_D1_TABLES.Stacked on #54
Migration numbers must be contiguous in each PR, so this branch stacks on
feat/canary-traffic(#54, migration 0025), which stacks onfeat/context-pages(#50, migration 0024). Merge those first.This branch starts from the glued Wave 4 base (cd4ac5f), so
integrate/gateway-waves-1-4was merged intofeat/context-pages, then down the stack. That keeps this PR's diff to the semantic cache. #50 and #54 were re-tested after that merge.Left for the Wave 5 integration
semantic-cachestate domain tohandleSemanticCacheinworker/control-state-d1.mjs, gated by the flag, with a 2 MiB limit. Until then, the Flask cache can't reach storage in a D1 deployment, and lookups fall through to normal dispatch.prepareSemanticCacheinnativeGenerationFetchwith the embedding transport, grant, budget and accounting collaborators.cleanupSemanticCacheto the scheduled maintenance.Tests
tests/test_semantic_generation_cache.pyandtests/test_semantic_generation_cache_worker.mjscover:n > 1, retention,no-store, unknown price, scopes that aren't opted in);npm run test:worker: 1,524 passed on a solo run (two timing-sensitive tests failed while three CI runs shared the machine).npm run test:integration: 7 passed.migrations:check(26 contiguous, additive migrations),test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.Apply the migration before enabling the flag:
npx wrangler d1 migrations apply multillm-intelligence --remote.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 5. The README line comes with the Wave 5 integration PR.